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Abstract 


The  Infrared  (IR)  Eye  was  developed  with  support  from  the  National  Search  and 
Rescue  Secretariat  (NSS),  in  view  of  improving  the  efficiency  of  airborne  search-and- 
rescue  operations.  The  IR  Eye  concept  is  based  on  the  human  eye  and  uses 
simultaneously  two  fields  of  view  to  optimize  area  coverage  and  detection  capability. 
It  integrates  two  cameras:  the  first  camera,  with  a  wide  field  of  view  of  40°,  is  used 
for  search  and  detection  while  the  second  one  has  a  narrower  field  of  view  of  10°, 
giving  the  higher  resolution  required  for  identification;  this  narrower  field  is  mobile 
within  the  wide  field  and  slaved  to  the  operator’s  line  of  sight  by  means  of  an  eye¬ 
tracking  system.  The  images  from  both  cameras  are  fused  and  shown  simultaneously 
on  a  high  resolution  CRT  display  unit,  interfaced  with  the  eye-tracking  unit  in  order  to 
optimize  the  human-machine  interface.  The  JR  Eye  system  was  flight  tested  using  the 
Advanced  System  Research  Aircraft  (Bell  412  helicopter)  from  the  Flight  Research 
Laboratory  of  the  National  Research  Council  of  Canada.  This  memorandum  describes 
the  prototype  and  its  design  approach,  presents  some  results  of  the  flight  tests, 
indicates  the  strengths  and  deficiencies  of  the  system,  and  suggests  future 
improvements  for  an  advanced  system. 


Resume 


L’Oeil  Infrarouge  (IR)  a  ete  mis  au  point  grace  au  soutien  du  Secretariat  national  de 
recherche  et  sauvetage  (SNRS).  pour  ameliorer  fefficacite  des  operations  aeriennes  de 
recherche  et  sauvetage.  Le  concept  de  l’Oeil  IR  est  caique  sur  fceil  humain  et  utilise 
simultandment  deux  champs  visuels  afin  d’optimiser  l’etendue  de  la  surface  couverte 
et  la  capacite  de  detection.  Deux  cameras  sont  integrees  :  la  premiere  est  utilisee  pour 
la  recherche  et  la  detection,  avec  un  champ  de  40°,  tandis  que  la  seconde  poss£de  un 
champ  de  10°,  plus  etroit  pour  une  plus  grande  resolution  necessaire  h  V identification; 
ce  champ  plus  Etroit  est  mobile  dans  le  cadre  du  champ  large  et  asservi  a  la  direction 
de  regard  de  1’operateur  sur  I’affichage  au  moyen  d’un  oculometre.  Les  images  des 
deux  cameras  sont  fusionnees  Tune  dans  l’autre  et  affichdes  simultandment  sur  un 
ecran  CRT  h  haute  resolution,  interface  avec  Toculomfetre  pour  une  interaction 
humain-machine  optimale.  L’Oeil  IR  a  subi  un  premier  test  en  vol  a  bord  de 
l’hdlicoptere  Bell  412  de  recherche  sur  les  systemes  evolues  du  Laboratoire  de 
recherche  en  vol  du  Conseil  national  de  recherches  Canada.  Ce  memorandum  decrit  le 
prototype  et  la  demarche  suivie  pour  sa  conception,  presente  quelques  resultats  du  test 
en  vol,  souligne  ses  forces  et  ses  faiblesses,  et  suggere  des  ameliorations  pour  un  futur 
syst^me  plus  avance. 
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Executive  summary 


The  purpose  of  Infrared  Eye  is  to  improve  night  and  adverse  weather  capabilities 
during  general  surveillance  and  search-and-rescue  (SAR)  missions,  not  only  by  using 
infrared  (ER)  or  thermal  imagers,  but  by  providing  a  system  which  is  better  adapted  to 
mission  and  user’s  requirements.  To  conduct  such  missions  efficiently,  particularly  in 
airborne  scenarios,  a  wide  area  coverage  is  required  to  maintain  the  situation 
awareness  of  the  operator  at  all  times,  while  reducing  the  search  time.  The  operator 
also  wants  a  high-resolution  capability  to  be  able  to  identify  detected  objects.  Those 
requirements  are  incompatible  when  using  a  single  camera  with  conventional  optical 
systems,  because  a  wide  field  of  view  generally  means  a  lower  resolution,  which  is 
detrimental  to  the  task  of  identifying.  Also,  as  such  missions  may  last  several  hours, 
with  a  high  degree  of  concentration  on  the  operator’s  part,  the  user-interface  must  be 
optimized  to  avoid  unnecessary  fatigue.  The  Infrared  Eye  concept  was  developed  to 
answer  those  special  requirements.  Following  a  first  laboratory  prototype,  a  second 
one  was  designed  to  experiment  the  concept  from  an  airborne  platform,  in  realistic 
scenarios. 

The  IR  Eye  concept  is  based  on  the  human  eye  and  uses  simultaneously  two  fields  of 
view  to  optimize  area  coverage  and  detection  capability.  It  integrates  two  cameras:  the 
first  camera,  with  a  wide  field  of  view  (WFOV)  of  40°,  is  used  for  search  and 
detection  while  the  second  one  has  a  narrower  field  of  view  (NFOV)  of  10°,  giving  the 
higher  resolution  required  for  identification;  this  narrower  field  is  mobile  within  the 
wide  field  and  slaved  to  the  operator’s  line  of  sight  by  means  of  an  eye-tracking 
system.  The  images  from  both  cameras  are  fused  and  shown  simultaneously  on  a  high 
resolution  CRT  display  unit,  interfaced  with  the  eye-tracking  unit  in  order  to  optimize 
the  human-machine  interface. 

The  IR  Eye  system  was  flight  tested  using  the  Advanced  System  Research  Aircraft 
(Bell  412  helicopter)  from  the  Flight  Research  Laboratory  of  the  National  Research 
Council  of  Canada. 

This  memorandum  describes  the  prototype  and  its  design  approach,  presents  some 
results  of  the  flight  tests,  indicates  the  strengths  and  deficiencies  of  the  system,  and 
suggests  future  improvements  for  an  advanced  system.  Following  the  flight  test, 
corrective  actions  were  taken  to  solve  some  technical  and  user  interface  deficiencies. 
Those  corrections  are  also  described. 

Although  the  WFOV  detection  capability  for  point  target  was  limited,  as  expected,  the 
flight  has  demonstrated  the  absolute  necessity  and  usefulness  of  the  WFOV  image  for 
the  situation  awareness  and  orientation  of  the  operator  with  respect  to  the  observed 
scene  from  the  helicopter.  The  NFOV  image  alone  was  insufficient  to  make  a 
correlation  with  the  visually  observed  scene  and  to  quickly  find  back  the  target  when  it 
went  off  the  NFOV  image  part. 
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Sommaire 


L’objectif  de  FOeil  Infrarouge  (IR)  est  d’ameliorer  la  capacite  d’ operation  la  nuit  ainsi 
que  dans  des  conditions  atmosphdrtques  adverses  lors  des  missions  de  surveillance 
gendrale  et  de  recherche  et  sauvetage,  non  seulement  par  F  utilisation  de  cameras  a 
infrarouge  (appareil  d’imagerie  thermique),  mais  aussi  en  integrant  un  systeme  mieux 
adaptd  aux  besoins  des  missions  et  de  Futilisateur.  Pour  mener  efficacement  de  telles 
missions,  particulierement  d’une  plate-forme  aeroportee,  il  est  necessaire  de  couvrir 
une  grande  surface,  tant  pour  maintenir  la  conscience  de  la  situation  de  Fopdrateur  en 
tout  temps  que  pour  diminuer  le  temps  de  recherche.  L’opdrateur  requiert  egalement 
suffisamment  de  resolution  pour  pouvoir  identifier  les  objets  detectes.  Ces  exigences 
sont  incompatibles  lorsqu’on  utilise  une  seule  camdra  avec  un  systdme  optique 
conventionnel,  parce  qu’un  champ  visuel  large  implique  une  moins  bonne  rdsolution, 
ce  qui  est  nefaste  k  F  identification.  De  plus,  comme  ces  missions  durent  plusieurs 
heures  et  exigent  une  grande  concentration  de  la  part  de  Fopdrateur,  F  interface  usager 
doit  etre  optimale  afin  de  rdduire  la  fatigue.  Le  concept  de  l’Oeil  IR  a  dtd  mis  au  point 
pour  repondre  a  ces  exigences  particulieres.  Apres  un  premier  prototype  de  laboratoire, 
un  second  a  etd  conqu  pour  experimenter  le  concept  k  partir  d’une  plate-forme 
aeroportde,  dans  des  scenarios  rdalistes. 

Le  concept  de  FOeil  IR  est  base  sur  Foeil  humain  et  utilise  simultanement  deux 
champs  visuels  pour  optimiser  a  la  fois  la  surface  couverte  et  la  capacite  de  detection. 
Deux  camdras  sont  integrdes  :  la  premiere,  avec  un  grand  champ  visuel  (WFOV)  de 
40°,  est  utilisee  pour  la  recherche  et  la  ddtection,  tandis  que  la  seconde,  avec  son 
champ  plus  etroit  de  10°,  possede  la  haute  rdsolution  necessaire  k  F  identification.  Ce 
champ  etroit  est  mobile  k  Finterieur  de  grand  champ,  et  asservi  a  la  direction  de  regard 
de  Foperateur  sur  Faffichage  par  Fintermddiaire  d’un  oculometre.  Les  images  des 
deux  camdras  sont  fusionndes  et  affichees  simultandment  sur  un  dcran  a  haute 
rdsolution,  auquel  est  coupld  un  oculometre  afin  d’ optimiser  F  interface  usager, 

L’Oeil  IR  a  dtd  teste  en  vol  sur  Fhelicoptdre  Bell  412  de  recherche  sur  les  systemes 
evolues  du  Laboratoire  de  recherche  en  vol  du  Conseil  national  de  recherches  Canada. 

Ce  mdmorandum  ddcrit  le  prototype  et  la  demarche  suivie  lors  de  sa  conception, 
presente  quelques  resultats  du  test  en  vol,  indique  les  forces  et  les  faiblesses  du 
systeme  et  suggere  les  ameliorations  k  apporter  pour  un  systeme  plus  evolue.  A  la  suite 
des  tests,  des  mesures  ont  dtd  prises  en  vue  de  corriger  certains  problemes  techniques 
et  d’ interface  usager.  Ces  correctifs  sont  discutes. 

Meme  si  comme  prevu,  la  capacite  de  ddtection  de  sources  ponctuelles  dtait  limitee 
dans  le  WFOV,  le  vol  a  demontre  la  necessite  et  Futilite  de  Fimage  WFOV  pour 
preserver  la  conscience  de  la  situation  de  Foperateur  et  son  orientation  par  rapport  k  la 
scdne  telle  que  vue  de  Fhelicoptere.  L’ image  NFOV  a  elle  seule  ne  permettait  pas  de 
faire  une  correlation  avec  la  scene  observee  visuellement  et  de  retrouver  rapidement  la 
cible  lorsqu’elle  sortait  de  ce  champ  visuel. 


IV 
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1 .  Introduction 


The  purpose  of  the  Infrared  Eye  is  to  improve  night  and  adverse  weather  capabilities 
during  general  surveillance  and  search-and-rescue  (SAR)  missions,  not  only  by  using 
infrared  (IR)  or  thermal  imagers,  but  by  providing  a  system  which  is  better  adapted  to 
mission  and  user’s  requirements.  To  conduct  such  missions  efficiently,  particularly  in 
airborne  scenarios,  a  wide  area  coverage  is  required  to  maintain  the  situation 
awareness  of  the  operator  at  all  times,  while  reducing  the  search  time.  However,  the 
operator  also  wants  a  high-resolution  capability  to  be  able  to  identify  objects  detected. 
Those  requirements  are  incompatible  when  using  a  single  camera  with  conventional 
optical  systems,  because  a  wide  field  of  view  generally  means  a  lower  resolution, 
which  is  detnmental  to  the  task  of  identifying.  Also,  as  such  missions  may  last  several 
hours,  with  a  high  degree  of  concentration  on  the  operator’s  part,  the  user-interface 
must  be  optimized  to  avoid  unnecessary  fatigue.  The  Infrared  Eye  concept  was 
developed  to  answer  those  special  requirements.  A  first  laboratory  prototype  was 
developed  to  explore  the  feasibility  of  the  basic  concept  [1].  Based  on  the  lessons 
learned  during  the  first  development,  a  second  prototype  was  designed,  to  solve  the 
deficiencies  of  the  first  one  and  to  experiment  the  concept  from  an  airborne  platform, 
in  realistic  scenarios.  This  memorandum  describes  the  latest  system,  shows  some 
results  from  the  first  flight  tests  and  explores  future  developments. 

This  work  was  carried  out  at  DRDC  Valcartier  between  April  2000  and  April  2002 
under  Thrust  3D  (Airborne  Search  &  Localization  Systems),  project  13DA  (Day  and 
Night  Search  Missions),  and  under  the  New  Search  &  Rescue  Initiative  Funds  (NIFID: 
98037)  program  from  the  National  Search  &  Rescue  Secretariat  (NSS). 

2.  Description  of  the  Infrared  Eye  concept 

The  IR  Eye  is  a  new  concept  of  search  and  rescue  (SAR)  and  surveillance  system.  It  is 
based  on  the  human  eye,  with  simultaneous  wide  (WFOV)  and  narrow  (NFOV)  fields 
of  view,  individually  enhanced  for  sensitivity  and  resolution,  respectively,  for  their 
specific  task  of  detection  and  recognition/identification  [2,3],  The  WFOV  spans  40° 
horizontally,  while  the  NFOV  covers  10°  and  is  mobile  within  the  WFOV.  A  special 
display  concept,  better  adapted  to  the  human  vision,  uses  a  very-high-resolution  CRT 
display  for  the  fusion  of  the  images  from  both  the  WFOV  and  NFOV  cameras.  An 
eye-tracking  system  determines  the  gaze  direction  of  the  operator  on  the  display  to 
steer  the  field  of  view  of  the  NFOV  camera  and  adequately  overlay  its  high-resolution 
image  on  the  WFOV  image  on  the  high-resolution  display  monitor,  aligned  with  the 
operator’s  central  vision. 

The  resolution  was  prioritized  over  the  waveband  for  the  selection  of  the  cameras,  in 
order  to  optimize  the  detection  capability  in  the  WFOV.  The  IR  camera  with  the 
highest  resolution  that  could  be  found  on  the  market  was  a  Mitsubishi  IR-M700,  with  a 
PtSt  focal  plane  array  (FPA)  of  800  x  512  pixels.  The  camera  operates  in  the  3-5  pm 
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band.  Ideally,  the  8-12  pm  band  would  be  preferable  for  the  envisaged  applications, 
but  no  high-resolution  FPA  was  commercially  available  at  the  time.  Two  IR  M700 
cameras  are  used  in  the  system  for  the  WFOV  and  the  NFOV,  respectively  (Figure  1 ). 


Figure  1  -  IR  Eye  optical  head  with  WFOV  and  NFOV  IR  cameras  and  Risley  prisms 


The  field-of-view  of  the  NFOV  camera  is  steered  by  means  of  an  opto-mechanical 
pointing  system  made  of  two  80-mm  diameter  achromatic  Risley  prisms, 
silicon/germanium  doublets  specifically  designed  for  the  application  by  the  National 
Optics  Institute,  Quebec  [4].  NOI  also  designed  the  NFOV  58-mm,//1.3  objective  lens 
with  a  ±  10%  zoom  capability,  to  allow  for  the  adjustment  of  the  4: 1  ratio  between  the 
two  FOVs.  The  prisms  are  mounted  coaxially  with  the  camera  objective,  which  avoids 
the  folded  path  and  image  rotation  introduced  by  a  two-axis  moving  mirror  (as  in  the 
first  prototype),  thus  making  the  system  more  compact  and  less  sensitive  to  vibrations, 
an  important  criterion  for  airborne  applications. 

The  4: 1  ratio  between  the  two  FOVs  imposes  constraints  on  the  display  system  since 
the  NFOV  image  must  be  properly  positioned  and  overlaid  with  scale  matching  on  the 
WFOV  image.  Since  the  NFOV  image  is  800  x  5 12  pixels  and  represents  (4  of  the 
horizontal  WFOV,  the  display  must  be  capable  of  3200  x  2048  pixels.  No  such  high- 
resolution  display  could  be  found.  However,  the  DEX2105L,  available  from  Clinton 
Electronics  (Orwin),  can  display  2560  x  2048  pixels.  It  is  a  bulky  CRT  specially 
designed  for  airborne  operation.  By  limiting  the  frame  grabbing  from  the  IR  M700 
cameras  to  640  x  5 12  pixels,  this  display  was  suitable  for  our  prototype. 
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For  positioning  the  NFOV,  three  eye-tracking  systems  were  tested  (Figure  2).  The  first 
two  are  models  501  and  504  from  Applied  Science  Laboratories.  The  third  system,  our 
final  choice,  is  model  RK-726PCI  from  Iscan  inc. 

Both  ASL  models  use  the  bright-pupil/comeal  reflection  principle  and  can  be 
combined  with  a  magnetic  head-tracking  system.  Model  501  is  a  head-mounted 
system,  while  model  504  is  a  desk-mounted  pan-tilt  camera,  thus  eliminating  all  head- 
mounted  apparatus.  Those  systems  are  good  for  psychometric  experiments  in  the 
laboratory,  but  the  magnetic  head-tracker  to  be  used  in  conjunction  with  them  is 
unsuitable  for  operation  in  aircraft  environment.  Model  504  accepts  head  movements 
within  a  cubic  foot,  without  the  head-tracking  system,  but  unfortunately,  the  tracking 
speed  of  the  pan-tilt  device  is  insufficient,  and  often  times  the  system  looses  track 
when  the  head  moves  too  fast,  even  within  the  allowed  cubic  foot. 

The  Iscan  system  met  our  expectations  with  an  optical  head-tracking  principle  and  a 
dark-pupil/comeal  reflection.  The  head-tracker  consists  of  6  light-emitting  diodes 
(LED),  mounted  on  the  frame  of  the  display  monitor,  and  a  small  filtered  camera  on 
the  head-mounted  device  (a  cap),  to  view  the  LEDs.  The  only  draw-back  so  far  is  the 
lack  of  some  library  functions  to  integrate  the  system  to  the  IR  Eye  control  software. 
Iscan  is  currently  developing  this  library.  The  IR  Eye  NFOV  can  be  steered  either  with 
a  mouse  click  at  the  desired  position  on  the  WFOV  image,  or  using  the  eye-tracking 
system. 

The  next  section  describes  in  more  details  the  opto-mechanical  pointer  driving  the 
Risley  prisms,  which  is  really  the  heart  of  the  IR  Eye. 


Figure  2  -  Eye-tracking  systems 
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3.  Opto-mechanical  NFOV  pointer 

3.1  The  eye  in  the  control  loop 

To  understand  the  challenge  designing  the  high-resolution  NFOV  steering 
mechanisms,  we  have  to  understand  how  the  operator’s  gaze  direction  (central  vision) 
is  used  to  move  the  high-resolution  area  of  the  image  (NFOV)  within  the  WFOV.  The 
human  eye  is  an  extremely  complex  sensor  but  fortunately,  to  build  the  steering 
mechanism,  we  only  need  to  consider  how  the  eye  moves  to  perceive  objects  and  its 
ability  to  resolve  moving  scenes.  The  human  eye  angular  rate  of  turn  is  more  than 
twice  faster  than  any  mechanical  system  we  could  build  to  steer  the  camera  NFOV. 
However  knowing  that  the  brain  takes  up  to  130  msec  [5, 6, 7, 8]  to  resolve  the  image 
when  the  ocular  globe  comes  to  a  stop,  we  can  use  this  time  to  stabilize  the  prisms  to 
their  final  positions  and  grab  the  image. 

As  mentioned  above,  an  eye-tracking  device  is  used  to  find  where  the  operator  gazes 
on  the  monitor.  The  computer  analyzes  the  eye  motions  and  moves  the  NFOV 
accordingly.  However  some  filtering  must  be  done  on  the  data  from  the  eye-tracking 
device  to  remove  the  eye  saccadic  motion  This  is  a  high-speed  random  motion  of  the 
ocular  globe  to  extract  the  high-resolution  details  of  a  scene  and  to  compensate 
instability  in  the  neuromuscular  system  of  the  eyes  [9].  The  saccadic  motion  is 
performed  on  a  relatively  small  portion  of  the  foveal  (central  and  high-resolution) 
region  of  the  eye  but  at  a  speed  that  the  mechanical  system  cannot  withstand.  To 
minimize  the  motion  of  the  NFOV,  we  have  conducted  tests  on  the  system  to 
determine  the  largest  “saccadic  motion  zone”  tolerable  without  seeing  the  borders  of 
the  NFOV  zone.  Our  experimentation  gave  a  “no-motion  radius”  of  3°  around  the 
center  of  the  NFOV,  that  is,  if  the  eye  moves  within  this  zone,  the  NFOV  position 
remains  stable.  However,  we  still  experience  sporadic  motion  of  the  NFOV  when  the 
eye  pokes  around  outside  this  zone.  We  control  this  behavior  by  adding  a  second  zone 
between  3°  and  5°  radius,  in  which  we  move  the  NFOV  halfway  toward  the  eye  gaze 
point  whenever  it  falls  into  that  zone  (Table  1). 


Table  1  -  Narrow  Field  of  View  displacement  strategy 


Eye  displacement  from  the  center  of  the  NFOV 

Displacement  of  the  NFOV 

0°  to  3° 

No  move 

3°  to  5° 

0.5  x  the  eye  displacement 

Greater  than  5° 

Reposition  the  NFOV  centered  on  the  eye  gaze 
point 

With  this  strategy,  we  achieve  a  comfortable  motion  of  the  high-resolution  NFOV 
without  undue  mechanical  constraints.  The  following  section  presents  the  different 
mechanical  concepts  we  tried  for  steering  the  NFOV,  based  on  the  above  precepts. 
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3.2  Description  of  mechanical  mounting  and  driving 


Numerous  concepts  have  been  studied  to  steer  the  NFOV  around  in  the  WFOV.  Two 
main  approaches  have  been  retained:  gimbaled  mirrors  and  Risley  prisms.  With  the 
first  beam  steering  technique,  the  NFOV  is  acquired  through  a  gimbaled  mirror  and  a 
de-rotator  prism  (first  prototype  of  the  IR  Eye).  The  prism  is  rotated  in  the  direction 
opposite  to  the  elevation  axis  to  compensate  for  the  image  rotation  around  the  optical 
axis.  The  challenge  in  this  system  is  more  in  the  optical  alignment  of  the  components 
than  in  the  concept  itself,  particularly  the  de-rotator  prism.  With  a  system  in  the  visible 
range,  we  could  use  a  laser  to  center  the  prism  on  the  optical  axis  but  in  the  3-5j0.m,  we 
had  to  use  an  IR  point-source,  seen  through  all  the  optical  components  and  use  the 
resulting  image  from  the  camera  to  adjust  the  prism  holding  fixtures.  We  had  good 
result  in  the  laboratory  and  in  controlled  test  environment  [3],  The  weak  point  of  this 
prototype  is  the  need  for  numerous  adjustment  points  to  control  the  position  and  the 
orientation  of  the  de-rotator  prism  which  is  very  sensitive  to  vibrations.  It  is  also 
voluminous  and  heavy. 

The  initial  goal  of  the  current  phase  of  the  project  (second  prototype)  was  to  build  a 
ruggedized  airborne  prototype  capable  of  withstanding  harsh  vibration  and  climatic 
conditions.  To  achieve  this  goal,  we  designed  a  new  steering  mechanism  with  a 
minimum  number  of  parts  and  adjustment  points.  As  mentioned  before,  the  pointing 
system  is  built  around  two  counter-rotating  achromatic  prisms,  known  as  Risley 
prisms.  Risley  prisms  consist  of  two  identical  round  wedges  (in  fact,  each  wedge  is  a 
Ge  and  Si  doublet)  mounted  on  independent  rotation  mechanisms.  Each  prism  deflects 
the  image  of  the  NFOV  by  a  fix  angular  vector:  in  our  case  by  10.5°.  By  rotating  each 
prism  to  a  specific  angle,  it  is  possible  to  reach  any  point  in  a  ±21°  space.  An 
advantage  of  Risley  prisms  is  that  they  offer  excellent  mechanical  stability,  contrary  to 
the  mirror  system,  do  not  induce  image  rotation,  and  any  constant  angular  error  due  to 
mechanical  misalignment  or  imperfections  can  be  compensated  for  at  the  calibration 
phase  of  the  system  (see  3.3).  The  rotation  mechanism  is  composed  of  the  prism  holder 
that  contains  the  Si-Ge  doublet,  3  pairs  of  delrin  rollers,  gears,  a  120W  brushless 
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motor  and  a  10,000  pulses/revolution  encoder.  One  of  the  design  challenges  was  to 
build  a  system  that  could  stay  accurate  and  survive  rotational  acceleration  up  to 
72,0007s2.  It  became  obvious  that  the  mechanical  components  should  withstand  huge 
accelerations  and  significant  wearing  without  getting  misaligned.  The  prism  holder  is 
supported  by  the  3  pairs  of  small  deinn  rollers  mounted  on  ball  bearings  on  each  side 
of  the  driving  gear.  Two  roller  pairs  are  fixed  to  support  the  pulling  force  of  the 
driving  belt  and  the  third  one  (at  the  opposite  side  of  the  motor)  is  spring  loaded  to 
control  the  pressure  of  the  rollers  on  the  holder.  The  prism  holder  is  correctly  aligned 
by  two  V  grooves  in  which  the  rollers  lean.  Weanng  could  change  the  diameter  of  the 
rollers  and  could  cause  an  eccentric  motion  of  the  holder  without  degradation  of  the 
pointing  precision.  The  current  prototype  has  been  used  for  more  than  a  year  without 
noticing  significant  degradation  of  the  pointing  accuracy  even  if  we  have  observed 
wearing  flats  on  the  delrin  rollers.  The  mechanics  is  designed  to  be  self-aligning 
(Figure  3). 

3.3  Pointing  calibration  procedure 

Although  mathematical  modeling  could  describe  perfectly  the  deviation  of  the  Risley 
prisms,  a  lack  of  precise  indexing  between  the  components  of  the  doublets  and  of 
mechanical  bore-sighting  accuracy  between  the  two  cameras  and  the  optical  axis  of  the 
pointing  system  led  us  to  develop  a  simple  and  effective  calibration  procedure  and 
software.  The  calibration  software  produces  a  lookup  table  of  the  position  of  the 
prisms  corresponding  to  the  deviation  angle  measured  from  the  center  of  displacement 
of  the  NFOV  in  the  WFOV. 

The  Risley  prisms  can  perform  two  basic  deviation  patterns  by  symmetric  motions  of 
the  prisms.  By  turning  both  prisms  in  the  same  direction  (coordinated  rotation),  the 
deviation  pattern  of  the  NFOV  in  the  WFOV  will  describe  concentric  circles  around  a 
center  of  alignment  corresponding  to  the  optical  axis  of  the  NFOV  camera  .  Symmetric 
counter  rotary  motion  of  the  prisms  gives  linear  motion  of  the  NFOV  through  the 
center  of  alignment.  Using  these  two  basic  motions,  we  can  direct  the  center  of  the 
NFOV  toward  any  point  in  the  WFOV  space.  Converting  the  Cartesian  pointing 
coordinate  in  the  WFOV  to  polar  coordinates  gives  us  the  radial  distance  of  a  point 
from  the  NFOV  center  of  alignment,  and  the  rotation  angle  from  an  arbitrary  origin 
(angle  of  rotation  on  a  concentric  circle).  The  calibration  process,  in  brief,  consists  in 
finding  the  correspondence  between  points  on  a  line  extending  from  the  center  of 
alignment  of  the  NFOV  in  the  WFOV  image  to  the  WFOV  border,  and  the  angular 
positions  of  the  steering  prisms.  The  calibration  process  also  removes  the  need  for 
precise  bore-sighting  between  the  NFOV  and  the  WFOV  cameras. 

To  calibrate  the  system,  the  camera  head  is  installed  sideway  on  a  pan  &  tilt  platform. 
The  tilt  axis  is  used  to  make  the  panning  plane  of  the  platform  parallel  to  the  horizontal 
axis  of  the  camera.  The  tilt  is  adjusted  in  such  a  way  that  a  distant  point-source  target 
(farther  than  100m)  appears  to  travel  on  a  horizontal  line  when  panning  the  camera 
head.  The  second  step  is  to  find  the  prism  index  positions  which  gives  the  minimum 
NFOV  deflection,  that  is  the  point  where,  for  a  coordinated  rotation  of  the  prisms,  the 
translation  movement  of  the  image  is  minimum.  In  fact,  with  the  prisms  at  those 
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angular  positions,  we  should  not  see  any  movement  of  the  image.  But  due  to  the  lack 
of  positioning  index  on  the  doublet  elements  and  the  mechanical  tolerances  in  the 
prism  design,  we  observe  a  motion  radius  of  3  pixels  in  the  NFOV  with  the  current 
system,  representing  a  pointing  error  of  1.6  mrad  in  the  worst  case.  Now,  knowing  the 
no-deflection  prism  positions,  we  move  the  camera  head  for  the  point  source  target  to 
appear  at  the  NFOV  image  center  and  mark  the  corresponding  point  in  the  WFOV, 
which  corresponds  to  the  center  of  alignment  of  the  NFOV  and  its  center  of  rotation 
for  all  displacements  with  the  prisms.  At  this  point,  the  images  of  the  cameras  should 
be  more  or  less  bore-sighted.  The  third  step  is  to  turn  the  camera  head  by  half  a  degree 
increments,  to  mark  the  target  image  in  the  WFOV  and  rotating  the  prisms  to  bring  the 
target  image  in  the  center  of  the  NFOV.  Recording  those  points  build  the  calibration 
table  of  the  system.  The  points  are  all  on  a  honzontal  line,  which  corresponds  to  the  0° 
axis  of  the  coordinated-rotation  angle. 

To  move  the  NFOV  to  a  point  in  the  WFOV,  we  calculate  the  polar  coordinate  of  this 
point  with  respect  to  the  NFOV  center  of  alignment,  we  find  the  radius  of  the  point  in 
the  calibration  table  (linear  interpolation  is  used)  and  we  add  to  the  corresponding 
prisms  position,  the  coordinated  rotation  angle  to  reach  the  point  in  the  WFOV. 


4.  Control  software  and  user  interface 


4.1  Crew  station  interface 

In  September  2001,  we  flew  with  version  4.0  of  the  crew-station  human-machine 
interface  software  (HMI).  Except  for  problems  found  in  the  acquisition  thread,  the 
main  improvement  from  the  previous  versions  was  a  simplified  control  interface.  From 
previous  ground-based  experiments,  we  determined  which  controls  were  essential  to 
the  mission  progress  as  opposed  to  system  management  and  diagnostics.  System 
management  and  diagnostics  have  been  moved  to  applications  specific  to  these  tasks. 
The  control  functions  are  found  in  three  groups  (Figure  4). 

The  first  group  takes  care  of  the  image  controls.  The  cameras  provide  us  with  10-bit 
images  from  which  we  select  a  range  to  be  fitted  to  the  256  gray  levels  of  the  high- 
resolution  monitor.  This  range  is  controlled  by  a  motion  of  the  mouse  on  the  screen 
and  three  on-screen  sliders.  An  up  and  down  motion  of  the  mouse  increase  or  decrease 
the  span  (or  the  number  of  gray  level  from  the  camera).  A  leftward  or  rightward 
motion  of  the  mouse  change  the  offset  of  the  image  (or  the  smallest  gray  level 
presented).  On-screen  sliders  can  also  be  used  to  select  and  visualize  the  lowest  and 
the  highest  gray  level  presented.  An  on-screen  button  allows  to  switch  the  control 
action  from  the  NFOV  to  the  WFOV.  The  second  group  contains  the  recording 
functions.  The  first  recording  functions  subgroup  controls  still-image  capture  (freeze 
image  and  save  to  disk).  The  second  recording  functions  subgroup  controls  the  live 
image  recording  and  playback.  As  on  a  VCR,  we  find  the  Record,  Play  and  Stop 
buttons.  The  Fast  forward  and  Rewind  buttons  in  addition  to  skip  in  time  could  jump 
to  recorded  events  as  typed  memo,  target  detection  and  tagged  images.  The  images  and 
data  are  recorded  digitally  at  7  frames  per  second  for  mission  archiving.  Finally,  we 


TM  2002-171 


7 


grouped  all  together  the  displaying  of  system  and  mission  status.  In  a  box,  we  display 
the  mission  current  time,  the  recorder  timing  and  active  function.  In  following  version 
of  the  system,  we  will  also  display  the  aircraft  altitude,  heading  and  position  in  this 
zone. 
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Figure  4  -  Crew  Station  control  tool  bar 


4.2  Image  acquisition  interface 

The  image  acquisition  interface  controls  four  independent  threads  that  perform  the 
following  functions,  respectively:  1)  to  grab  images  from  the  frame-grabber  cards;  2) 
to  perform  eye-tracking  and  direction  of  regard  management;  3)  to  control  the  NFOV 
image  steering  mechanics;  4)  to  enhance  point-target  detection  in  view  of  helping  the 
SAR  operator  in  finding  small  and  faint  target  (in  development). 

5.  Flight  test 


5.1  Aircraft  installation 
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The  IR  Eye  was  flight-tested  in  cooperation  with  the  Flight  Research  Laboratory  from 
the  National  Research  Council  of  Canada’s  Institute  for  Aerospace  Research,  on  their 
Advanced  Systems  Research  Aircraft,  a  Bell  412  helicopter.  This  helicopter  is  similar 
to  the  Griffon,  owned  by  the  Canadian  Forces,  a  candidate  aircraft  to  eventually  fly  the 
IR  Eye  on  search-and-rescue  or  surveillance  missions.  At  this  stage  of  development, 
the  IR  Eye  packaging  was  not  quite  suitable  for  standard  flight  missions  and  required  a 
very  flexible  certification  and  operation  methodology,  which  only  FRL  could  supply. 

An  aluminum  cover  with  tilted  germanium  windows  was  installed  over  the  camera 
head  to  protect  it  against  dust  and  weather.  A  small  video  camera  operating  in  the 
visible  band  was  affixed  to  the  cover.  The  camera  head  was  mounted  on  a  Tyler 
Middle  Mount  II,  a  man  operated  airborne  stabilization  platform  used  in  the  cinema 
industry.  The  Tyler  Mount  was  strapped  to  the  helicopter  floor  behind  the  pilot  seat 
and  the  operator  could  sit  in  the  Tyler  mount  seat  with  feet  on  a  foot  support 
overhanging  the  open  cargo  side  door,  on  the  port  side  of  the  aircraft,  to  point  the 
camera  head  in  the  proper  direction  (Figure  5). 

A  special  floorboard,  made  by  FRL,  was  bolted  to  the  floor  on  the  starboard  side,  to 
serve  as  a  support  for  the  electronics  rack.  The  rack  itself,  of  soldered  aluminum  struts, 
was  based  on  an  approved  design  for  airworthiness  acceptance,  and  held  the  heavy  (85 
lbs)  high-resolution  display  monitor,  the  industrial  computer,  power  supplies  for  the 
cameras  and  the  prism  drive  motors,  and  two  video  recorders  for  a  backup  recording  of 
the  flight  test  (one  recorder  was  used  for  the  WFOV  IR  cameras  and  the  other  could  be 
switched  between  the  NFOV  and  the  visible  cameras).  The  keyboard  had  an  included 
joystick-type  mouse  and  was  mounted  in  a  drawer  so  that  it  could  be  moved  out  of  the 
way  during  take-offs  and  landings. 

The  aircraft  was  equipped  with  an  inverter  that  could  supply  1 10V760Hz,  10A,  from 
the  28VDC,  which  was  amply  sufficient  power  for  our  system.  The  inverter  is  also  a 
standard  equipment  on  the  CF  Griffon, 

5.2  Airborne  results,  first  flight 

The  first  flight  was  scheduled  for  September  12,  2001,  with  the  participation  of  a 
SARtech  (search-and-rescue  technicians)  team  from  Trenton,  Ontario,  CA.  The  plan 
was  to  fly  from  Ottawa  to  Trenton  and  to  simulate  SAR  scenarios  on  Lake  Ontario, 
using  small  boats  and  including  a  man  in  water,  with  the  cooperation  from  the 
SARtechs.  Unfortunately,  because  of  the  September  11  events,  we  were  grounded  until 
the  13th,  noon  time,  and  the  participation  of  the  SAR  team  was  canceled.  Our  flight 
was  limited  to  some  specific  training  areas  near  Ottawa.  The  objectives  of  that  first 
flight  were  to  determine  the  operability  of  the  system  in  a  real  helicopter  environment 
and  the  efficiency  of  the  stabilization  with  the  Tyler  Mount,  to  evaluate  the  effect  of 
the  low-frame-rate  display  (about  7  fr/s)  with  moving  scenes,  and  finally  to  do 
preliminary  performance  tests  with  a  someone  on  the  ground  in  a  semi-forested  area. 

Figures  6  to  8  show  images  as  presented  on  the  high-resolution  display.  Those  images 
are  composed  by  blowing  the  WFOV  image  (640  x  5 12  pixels)  by  a  factor  of  4  x  4  to 
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fill  the  2560  x  2048  pixels  of  the  display  screen,  and  then  inserting  the  NFOV  image  in 
the  proper  position  and  in  its  normal  dimensions  (640  x  512),  to  preserve  the  relative 
angular  coverage  of  each  FOV.  The  difference  in  resolution  capability  between  the 
two  FOVs  is  obvious.  Defects  are  also  apparent,  showing  errors  in  magnification  and 
sometimes  in  position.  On  some  images,  interlace  problems  are  apparent  as  either  an 
inversion  of  frame  fields  or  a  field  separation  caused  by  rapid  movements  of  the  scene 
between  the  field  grabbing. 

The  following  preliminary  observations  were  made  after  the  flight  test: 

•  The  stabilization  with  the  Tyler  mount  is  adequate,  but  its  use  is  strenuous  on  the 
operator  because  of  wind  effect.  Its  open-door  operating  requirement  is 
incompatible  with  the  real  all-year-round  search-and-rescue  operational 
environment. 

•  The  major  drawback  was  the  splitting  between  the  interlaced  fields  composing  the 
WFOV  images  when  the  scene  is  moving  and  while  digital  sequences  are  recorded 
(which  slows  the  processor).  This  problem  does  not  seem  to  occur  in  the  NFOV. 
This  is  a  software  problem  caused  by  the  interlace  format  of  the  digital  video 
output  from  the  cameras  and  by  a  heavy  workload  on  the  processor.  Also, 
sometimes  the  order  of  the  fields  seems  to  be  inverted,  which  is  also  a  software 
problem  caused  by  the  incompatibility  of  the  video  output  from  the  cameras  with 
the  acquisition  cards. 

•  The  low  frame-rate  (7  fr/s)  seems  tolerable  to  the  operator,  as  the  steps  between 
images  is  a  small  portion  of  the  FOV,  as  long  as  a  full  frame  is  grabbed  with  two 
consecutive  fields. 

•  The  in-flight  adjustment  of  contrast  and  brightness  for  both  cameras  (crew  station 
interface)  needs  improvement.  Using  an  airborne  intended  system  on  ground  led 
to  misevaluating  parameters  specific  to  airborne  platforms  like  vibrations,  strong 
and  changing  translational  and  rotational  accelerations  (in  3D),  fast  changing 
illumination  conditions  and  restricted  work  space.  We  found  the  screen-based 
user-interface  and  the  on-screen  buttons  difficult  to  use  and  disturbing.  At  design 
time,  we  took  a  special  care  in  using  the  simplest  and  stable  pointing  device  (Hula 
Point  joystick  on  a  keyboard,  model  5000  from  Stealth  Computer)  and  in  finding 
the  most  effective  way  to  use  it,  but  the  flight  conditions  were  worst  than  what  we 
were  expecting  to  deal  with.  This  kind  of  pointing  device  and  on-screen  user 
interface  is  not  really  effective  and  disturbs  the  operator  from  its  main  task 
namely:  searching  for  targets,  rather  than  following  the  Windows  mouse  pointer! 
In  fact,  with  this  type  of  user  interface,  three  operations  must  be  performed: 
finding  the  pointer  on  the  screen,  finding  the  on-screen  button  to  click  and  moving 
the  pointer  to  the  on-screen  button.  It  seems  simple,  but  in  flight  condition,  all  the 
operator’s  attention  is  required  to  coordinate  those  operations  on  a  high-resolution 
display  where  cursor  and  button  icons  become  very  small,  which  makes  the 
operator  loose  track  of  the  mission  progress. 
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Figure  6  -  Image  of  a  parking  lot  as  captured  with  the  IR  Eye  system 
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Figure  7  -  View  of  a  freeway  exchanger  as  captured  with  the  IR  Eye  system 
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Figure  8  -  Cows  in  a  field,  as  captured  by  the  IR  Eye  system 
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•  The  digital  sequence  recording  software  did  not  work  properly.  Only  still, 
composed  digital  images  could  be  recorded  (as  viewed  on  the  high-resolution 
display  screen). 

•  The  4-in  LCD  display,  mounted  on  the  Tyler  mount  seat  to  help  the  operator  in 
pointing  the  camera  head,  was  inadequate  in  terms  of  contrast  and  viewing  angle 
and  its  mounting  position  was  not  ideal  for  the  operator’s  comfort. 

•  As  expected,  the  WFOV  did  not  perform  very  well  in  detection  mainly  for  three 
reasons: 


1 .  at  constant  //number  a  loss  of  aperture  occurs  when  optically  increasing 
the  FOV  on  a  sensor  by  changing  the  focal  length  of  the  objective.  In  our 
case  a  4: 1  ratio  between  the  WFOV  and  the  NFOV  results  in  a  16  time 
reduction  of  the  aperture  area,  with  an  equivalent  loss  in  detection 
capability; 

2.  at  lower  ambient  temperatures,  the  3-5  \xm  band  is  not  the  best  choice  for 
low-contrast  imaging  because  of  the  lower  available  signal  in  that  band; 

3.  the  contrast  between  the  man  and  the  background  was  further  reduced  by 
the  sun  heating  of  the  background. 

•  However,  the  flight  has  demonstrated  the  absolute  necessity  and  usefulness  of  the 
WFOV  image  for  the  situation  awareness  and  orientation  of  the  operator  with 
respect  to  the  observed  scene  from  the  helicopter.  The  NFOV  image  alone  was 
insufficient  to  make  a  correlation  with  the  observed  scene  and  to  quickly  find  back 
the  target  when  it  went  off  the  NFOV  image  part. 

AH  problems  and  deficiencies  were  noted  of  in  view  of  improving  the  system  for  other 
flight  tests  to  follow. 

6.  Corrective  actions  taken  following  the  flight  test 

After  the  first  flight,  the  equipment  was  brought  back  to  the  laboratory  and  the  effort 
was  directed  toward  correcting  the  noted  deficiencies. 

•  The  most  important  problem  was  the  apparent  splitting  between  the  interlaced 
fields  composing  the  images,  particularly  apparent  in  the  WFOV.  This  problem 
became  apparent  when  there  was  a  scene  motion  and  was  further  enhanced 
because  the  fields  did  not  seem  to  be  grabbed  consecutively;  it  was  also 
accompanied  sometimes  by  an  inversion  between  the  odd  and  even  fields.  As 
previously  mentioned,  this  problem  was  caused  by  a  frame-grabber  card  not 
designed  for  interlaced  digital  video  input  (Road  Runner  from  BitFlow)  Before 
the  flight  test,  an  attempt  was  made  to  correct  the  card  deficiency  by  software, 
grabbing  fields  separately  and  then  composing  the  image  into  the  graphic  memory 
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To  solve  this  problem,  tests  were 
performed  with  a  Meteor  card  from 
Matrox,  which  is  designed  to  accept 
interlaced  fields.  The  tests  were 
successful  and  we  are  in  the  process  of 
adapting  the  control  software  to 
integrate  Matrox  cards  in  replacement 
of  the  BitFlow  cards.  The  computer  was 
upgraded  to  a  dual  1  -GHz  processor 
board,  in  an  attempt  to  increase  the 
frame-grabbing  and  processing  speed. 

•  To  improve  the  user  interface,  we 
studied  some  options  like  “mechanical” 
buttons  and  sliders  on  the  screen  frame,  or  using  the  eye-tracking  device  with  on¬ 
screen  labels.  The  first  solution  is  frequently  found  in  moving  platform  like 
airplanes  or  tanks.  In  our  case,  this  solution  was  not  ergonomic  as  the  operator  is 
continuously  using  the  interface  and  his  arms  will  never  rest  on  something,  rapidly 
leading  to  operator  fatigue.  The  eye-tracking  device  could  be  a  nice  input  device 
for  the  operator  console  but  would  be  too  complex  to  use  and  could  be  unreliable. 
The  solution  adopted  is  a  knee-pad  of  function  keys  and  a  thumb  wheel.  The  keys 
are  positioned  to  be  used  without  looking  at  the  knee-pad  to  find  them.  This  type 
of  user  interface  allows  the  operator’s  hands  and  arms  to  rest  in  an  ergonomic 
position  and  to  use  the  system  with  convenience.  The  interface  will  be  completed 
before  our  next  flight  with  the  system  (Figure  9). 

•  The  corrections  to  software  for  the  digital  recording  of  sequences  are  nearly 
completed. 

•  Some  tests  were  carried  with  a  variety  of  head-mounted  micro-displays,  purchased 
within  another  DRDC  Valcartier  project,  in  view  of  using  those  as  a  viewing  aid 
for  the  operator  sitting  in  the  Tyler  Mount  and  pointing  the  cameras  to  the  target. 
The  micro-displays  will  be  made  available  for  our  next  flights. 

•  A  low-cost  in-house  stabilization  mount  has  been  designed  and  components  are 
being  tested  and  manufactured  at  our  machine  shop.  This  mount  will  be 
compatible  with  the  support  used  on  the  Bell  412  for  the  Wescam  ball  and  will 
allow  to  fly  with  the  cameras  mounted  outside  the  helicopter,  which  should  be  a 
better  adaptation  for  the  considered  applications.  The  IR  Eye  requires  a 
stabilization  precision  significantly  lower  than  what  the  Wescam  units  can  provide 
because  of  its  relatively  large  fields  of  view  and  the  tolerance  of  a  human  operator 
to  low-frequency  movements  of  the  platform. 

•  A  laboratory  characterization  of  the  complete  system  was  made.  This  evaluation 
includes  the  exact  spectral  response  of  the  camera,  the  spectral  transmission  of  the 
prisms,  the  resolution  (MTF),  distortions  and  the  NETD  figures  with  and  without 
the  prisms,  and  the  positioning  accuracy  of  the  opto-mechanical  system.  Other 
electronic  characteristics  such  as  the  slew-rate  and  the  optimum  frame-rate  will 
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also  be  measured,  when  software  and  hardware  configurations  will  be  finalized. 
This  will  help  in  understanding  and  predicting  the  performance  of  the  system  in 
future  tests  and  in  determining  technical  solutions  to  the  deficiencies  for  future 

systems. 

•  The  next  ground-based  and  flight  tests  will  concentrate  on  the  system  efficiency  in 
SAR  missions,  and  try  to  determine  detection  and  recognition  ranges  for  a  variety 
of  targets  and  scenarios,  in  various  types  of  backgrounds  and  weather  conditions. 

7.  Future  improvements 

As  the  development  of  the  IR  Eye  prototype  is  being  finalized  to  make  it  an  airworthy 
demonstration  system,  future  evolutions  are  envisaged  to  refine  its  efficiency  and 
usefulness. 

7.1  VIZIR 

The  VIZIR  project,  also  supported  by  NSS,  consists  in  adding  an  active,  range-gated, 
near-infrared  (NIR)  camera  in  the  NFOV  to  enhance  the  effectiveness  of  the  system 
and  take  advantage  of  the  NIR  technology  in  using  the  reflective  characteristics  of  the 
target.  For  instance,  thermal  IR  does  not  allow  to  read  letterings  on  a  ship  because 
most  paints  have  an  emissivity  close  to  one  in  the  thermal  IR,  irrespective  of  their 
visual  color,  which  reduces  to  nothing  the  emissive  contrast  in  the  thermal  IR  bands.  In 
the  NIR  band,  the  difference  in  reflectivity  between  different  color  paints  is  about  the 
same  as  in  the  visible  band  and  produces  a  readable  contrast  The  use  of  an  active 
system  (with  an  illuminator)  also  adds  the  possibility  of  detection  of  optical  systems, 
including  the  eyes  of  a  person,  through  the  cat-eye  effect,  or  a  retro-reflection  of  the 
illuminating  light  from  the  focal  plane  of  the  optical  system.  Finally,  the  range-gating 
capability  allows  to  penetrate  to  some  extent  (depending  on  the  power  of  the 
illuminator)  adverse  weather  conditions  such  as  fog,  rain  or  snow,  by  eliminating  the 
backscattering  from  the  optical  path  between  the  camera  and  the  target.  The  range  to 
the  target  is  obtained  as  a  by-product.  DRDC  Valcartier  has  developed  an  expertise  in 
active,  range-gated  cameras  [10].  The  VIZIR  system  will  let  the  operator  select  either 
the  IR  or  the  active  image  for  the  NFOV  portion  of  the  displayed  image,  and  to  zoom 
on  the  target  for  a  better  identification.  The  possibility  of  a  fusion  of  the  IR  and  active 
images  is  also  considered. 

A  laboratory  prototype  is  being  developed  at  DRDC  Valcartier  and  will  operate  in  the 
8-12  pm  IR  band  for  both  WFOV  and  NFOV.  An  active  NFOV  with  a  laser 
illuminator  at  0.855  pm  are  added.  The  laser  illuminator  diode  array  (28  W  at  15.75 
kHz  pulse  rate)  has  been  designed  and  manufactured  by  NOI.  The  Risley-pnsm 
approach  is  taken  for  all  moving  FOVs.  The  VIZIR  projects  wall  benefit  from  the 
expertise  developed  within  the  IR  Eye  project  for  the  control  software,  as  well  as  in  the 
design  and  fabrication  of  opto- mechanical  pointers. 
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7.2  Step-stare 


The  original  idea  behind  the  IR  Eye  concept  was  to  develop  a  system  which,  similarly 
to  the  human  eye,  could  show  as  good,  if  not  better,  detection  capability  in  the  WFOV 
than  in  the  NFOV.  The  human  eye  peripheral  vision  is  5  orders  of  magnitude  better  in 
sensitivity  than  in  its  central  vision,  which  makes  our  peripheral  vision  our  detection 
sensor,  drawing  our  attention  so  that  we  bring  our  central  vision  to  the  detection  area 
to  identify  the  cause  of  anomaly.  Of  course,  our  peripheral  vision  is  quite  complex, 
involving  sophisticated  processing  through  a  neural  network  onto  the  retina,  before  a 
signal  is  sent  to  the  brain.  For  SAR  as  well  as  general  surveillance  applications,  this  is 
the  kind  of  sensor  that  is  required,  to  cover  large  areas  with  a  low  risk  of  missing  a 
target.  Unfortunately,  conventional  optics  and  sensor  technology  do  not  allow  yet  to 
combine  high  point-source  (smaller  than  the  pixel  angular  coverage)  detection 
sensitivity  and  a  large  FOV.  Our  eyes  use  the  same  lens  aperture  for  both  our 
peripheral  and  central  vision,  which  guarantees  at  least  the  same  signal  amplitude  for  a 
given  point  source.  The  increase  in  FOV  is  obtained  by  a  larger  sensor  area.  With 
conventional  optics,  since  the  sensor  dimensions  are  limited,  we  increase  the  FOV  by 
reducing  the  focal  length  of  the  lens.  Since  the  f/number  is  fixed  by  the  sensor  cold- 
shield,  a  shorter  focal  length  results  in  a  smaller  aperture,  limiting  the  amount  of 
energy  captured  for  the  same  point  source,  therefore  limiting  also  the  signal  amplitude 
and  the  detection  capability. 

In  the  absence  of  large  area  focal  plane  arrays,  one  solution  is  to  build  the  WFOV  as  a 
mosaic  of  images  taken  with  the  NFOV  optics.  Our  Risley  prism  approach  allows  us  to 
do  that  elegantly,  without  moving  the  whole  camera.  For  optimum  operation  speed,  a 
snapshot  (instead  of  rolling  integration)  camera  is  required  to  minimize  the  loss  of 
frames  between  displacements  of  the  FOV,  but  also  to  minimize  image  distortion  due 
to  motion  of  the  scene  within  each  frame  when  each  line  is  integrated  sequentially. 

The  limiting  factor  would  be  the  slew-rate  of  the  prisms  to  properly  position  the 
NFOV,  but  since  the  desired  positions  are  fixed  and  adjacent  in  space,  the  positioning 
sequence  would  be  repetitive  and  could  be  made  to  minimize  the  relative  prism 
displacement  between  positions,  thus  optimizing  the  grabbing  speed  for  the  overall 
mosaic  WFOV.  Each  NFOV  image  composing  the  WFOV  being  a  snapshot  of 
relatively  short  integration  time,  the  scene  motion  blur  on  each  individual  snapshot 
would  be  minimal;  the  scene  displacement  between  snapshots  could  be  compensated 
on  the  display  by  accounting  for  the  displacement  from  navigational  information  and 
properly  positioning  the  snapshots  on  the  screen  in  a  puzzle-like  fashion. 

The  development  of  algorithms  to  accomplish  such  a  motion  compensation  will  be 
experimented  using  a  simulator  developed  by  LYRtech  Inc.  for  psychometric 
experiments  related  to  the  IR  Eye  concept.  Special  algorithms  will  also  be  studied  for 
pixel  compression  of  the  high-resolution  WFOV  image  obtained  by  step-stare,  while 
preserving  the  detection  capability.  The  full  resolution  would  be  presented  in  an  area 
around  the  operator  central  field  of  regard  on  the  display,  determined  by  means  of  the 
eye-tracker,  while  in  the  rest  of  the  image,  pixels  could  be  fused  into  larger  pixels  on 
an  intensity  basts  (instead  of  averaging)  to  preserve  the  detection  capability.  For 
example,  within  a  group  of  pixels,  the  maximum  intensity  could  be  determined,  and  a 
larger  pixel  with  that  maximum  intensity  could  be  presented  on  the  screen  in  lieu  of 
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the  group  of  pixels,  which  should  draw  the  attention  of  the  operator.  Eventually,  if  the 
overall  acquisition  frame-rate  is  sufficiently  high,  one  of  the  camera  could  be 
eliminated,  keeping  the  eye-tracking  system  to  determine  image  areas  with  different 
compression  and  processing  schemes. 


7.3  Pixelless  QWIP/LED 

A  second  solution  to  preserve  the  point-source  detection  sensitivity  in  the  WFOV  is  to 
fabricate  a  larger  sensor  in  order  to  capture  a  wider  FOV  with  the  same  lens  and  the 
same  aperture  as  that  of  the  NFOV.  DRDC  Valcartier  is  currently  pursuing  that 
approach  in  cooperation  with  the  Institute  for  Microstructural  Sciences  (IMS)  of  the 
National  Research  Council  of  Canada.  Dr.  H.C.  Liu  from  IMS  has  developed  a  new 
type  of  infrared  sensor  called  a  pixelless  quantum- well-infrared-photodetector 
coupled  with  a  light-emitting-diode  layer  (QWIP-LED)  [11].  Basically,  this  sensor  is  a 
converter  from  thermal  infrared  wavelengths  to  near-infrared  (=0.85  pm).  The  sensor 
is  based  on  GaAs  and  QWIP  technologies.  The  QWIP  part  is  made  of  superposed 
layers  of  GaAs  and  AlGaAs  materials.  The  thickness  of  the  layers  determines  the 
detector  waveband,  which  thus  can  be  tuned  over  a  fairly  broad  range  of  the  thermal 
IR  spectrum.  A  few  more  added  layers  constitute  a  LED  through  which  must  pass  all 
photoelectrons  generated  by  the  capture  of  infrared  photons  in  the  QWIP  part,  to  reach 
the  biasing  electrode.  In  the  process,  the  photoelectrons  are  converted  into  NIR 
photons.  The  device  is  quite  simple  of  fabrication  and  operation,  requiring  only  two 
wires  and  low  voltage  for  its  biasing.  Since  the  overall  thickness  is  of  the  order  of  2 
microns,  the  lateral  diffusion  of  photoelectron  is  small  and  it  is  not  necessary  to 
subdivide  the  sensing  area  into  individual  pixels,  the  excitation  and  photon  conversion 
remaining  local,  hence  the  “pixelless”  denomination.  The  resolution  is  then  mainly 
determined  by  the  size  of  the  blur  spot  from  the  front  optics.  Another  advantage  of  the 
simplicity  of  the  device  is  that  it  can  be  made  in  large  format.  We  are  aiming  at 
fabricating  a  4  cm  x  4  cm  IR  sensor,  to  cover  a  WFOV  of  40°  with  the  same  lens  focal 
length  as  we  are  currently  using  for  the  10°  NFOV.  The  light  emitted  from  the  LED 
side  is  imaged  onto  a  CCD  sensor,  sensitive  to  the  NIR  wavelengths. 

The  main  drawback  is  the  difficulty  in  coupling  out  the  NIR  light  from  the  LED 
because  of  the  high  index  of  refraction  of  the  GaAs  substrate  material.  It  is  planned  to 
use  a  fiber-optic  faceplate  in  close  proximity  to  the  LED  surface  to  increase  the 
external  coupling  efficiency. 

The  research  is  progressing  well  and  it  is  hoped  that  eventually  such  a  large  area 
device  will  constitute  the  heart  of  the  IR  Eye,  eliminating  the  need  for  two  cameras.  As 
in  the  step-stare  case,  special  processing  algorithms  could  be  used  to  compress  the 
imager  data  while  preserving  the  detection  capability  in  the  wide  field  and  the 
resolution  in  a  smaller  area  centered  on  the  operator’s  line  of  sight. 
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8.  Conclusion 


The  feasibility  of  the  IR  Eye  concept,  with  two  fields  of  view  operating 
simultaneously,  by  analogy  with  the  human  eye,  has  been  first  demonstrated  with  a 
laboratory  prototype.  A  second  prototype,  using  the  latest  IR  focal  plane  array  camera 
technology  for  a  smaller  and  lighter  system  with  a  higher  resolution,  was  built  to  test 
the  concept  in  more  realistic  search-and-rescue  and  surveillance  scenarios,  from  an 
airborne  platform.  The  system  was  further  improved  in  compactness  and  ruggedness 
by  using  Risley  prisms  instead  of  a  gimbaled  mirror  and  a  de-rotator  prism  to  steer  the 
field  of  view  of  the  NFOV  camera  within  the  field  of  the  WFOV  camera.  A  special 
doublet  prism  design  was  required  to  make  the  prisms  achromatic  within  the  ER 
waveband  of  the  cameras.  Similarly,  a  special  opto-mechanical  system  was  designed 
to  hold  and  rotate  the  prisms  with  the  accuracy  required  for  the  application.  The 
system  was  integrated  with  general  control  software  allowing  all  the  necessary  control 
functions  for  the  camera  adjustments,  the  reading  of  the  eye-tracking  system,  the 
positioning  of  the  prisms,  the  superpositioning  of  the  two  FOVs  on  the  high-resolution 
display  and  the  recording  of  composed  images.  Finally,  the  system  was  mounted  on  a 
helicopter  and  tested  during  a  first  flight  in  order  to  determine  its  airworthiness  and 
user-interface  deficiencies.  Corrective  actions  were  taken  following  the  flight  in  view 
of  improving  particularly  the  user  interface  for  controlling  the  cameras  and  for 
improving  the  overall  frame-rate  and  display  quality.  Future  flights  will  concentrate  on 
measuring  the  efficiency  improvements  in  SAR  scenarios. 

A  number  of  areas  have  been  identified  for  further  research  and  development  in  order 
to  extend  the  capability  of  the  IR  Eye  and  improve  its  sensitivity  and  performances. 
The  addition  of  a  range-gated  active  camera  is  already  in  progress  and  the  integration 
of  the  complete  system  will  begin  soon.  To  increase  the  sensitivity  and  detection 
capability  in  the  WFOV,  two  approaches  are  considered:  the  development  of  a  large- 
area  pixelless  QWIP-LED  sensor  could  lead  to  a  single  camera  to  perform  the  same 
tasks  currently  done  with  the  dual-camera  IR  Eye  system.  However,  since  the 
development  of  such  a  sensor  is  still  in  a  research  phase,  a  step-stare  approach  could 
be  an  acceptable  compromise,  by  building  the  WFOV  out  of  a  mosaic  of  NFOV  image 
and  appropriately  tiling  them  together  on  the  display  to  compensate  scene  motion.  It  is 
hoped  that  advances  in  technology  (e.g.  uncooled  IR  sensors)  will  allow  the 
miniaturization  of  the  system  and  its  reduction  in  size  and  weight,  thus  making  it  more 
practical  for  integration  to  small  airborne  stabilization  mounts.  The  IR  Eye  has  many 
other  application  possibilities  as  a  landbased  and  shipboard  system,  which  have  not 
been  explored  yet. 
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The  Infrared  (IR)  Eye  was  developed  with  support  from  the  National  Search-and-Rescue  Secretariat  (NSS),  in  view  of  improving  the  efficiency  of  airborne 
search-and-rescue  operations.  The  IR  Eye  concept  is  based  on  the  human  eye  and  uses  simultaneously  two  fields  of  view  to  optimize  area  coverage  and 
detection  capability.  It  integrates  two  cameras:  the  first,  with  a  wide  field  of  view  of  40°,  is  used  for  search  and  detection  while  the  second  camera  has  a 
narrower  field  of  view  of  10°,  giving  the  higher  resolution  required  for  identification;  this  narrower  field  is  mobile  within  the  wide  field  and  slaved  to  the 
operator’s  line  of  sight  by  means  of  an  eye-tracking  system.  The  images  from  both  cameras  are  fused  and  shown  simultaneously  on  a  high  resolution  CRT 
display  unit,  interfaced  with  the  eye-tracking  unit  in  order  to  optimize  the  man-machine  interface.  The  IR  Eye  system  was  flight  tested  using  the  Advanced 
System  Research  Aircraft  (Bell  412  helicopter)  from  the  Flight  Research  Laboratory  of  the  National  Research  Council  of  Canada.  This  report  describes  the 
prototype  and  its  design  approach,  presents  some  results  of  the  flight  tests,  Indicates  the  strengths  and  deficiencies  of  the  system,  and  suggests  future 
Improvements  for  an  advanced  system. 

L’Oeil  Infrarouge  (IR)  a  6te  mis  au  point  grfice  au  support  du  Secretariat  National  Recherche  et  Sauvetage  (NSS).  pour  ameiiorer  I’efficacite  des  operations 
a6rlennes  de  recherche  et  sauvetage.  Le  concept  de  I’oell  IR  est  caique  sur  I’ceil  humain  et  utilise  simultan6ment  deux  champs  de  vue  afin  d'optimiser 
l’6tendue  de  surface  couverte  et  la  capacite  de  detection.  Deux  cameras  sont  integrees  :  la  premiere  est  utilise©  pour  la  recherche  et  la  detection,  avec  un 
champ  de  40°,  tandis  que  la  seconde  possede  un  champ  de  10°,  plus  etroit  pour  une  plus  grande  resolution  necessaire  k  I’ identification;  ce  champ  plus 
etroit  est  mobile  dans  le  cadre  du  champ  large  et  asservi  k  la  direction  de  regard  de  I’operateur  au  moyen  d’un  systems  de  suivi  oculaire.  Les  Images  des 
deux  cameras  sont  fusionn6es  I’une  dans  (’autre  et  affich6es  simultanement  sur  un  6cran  CRT  k  haute  resolution,  interface  avec  le  systems  de  suivi 
oculaire  pour  une  interaction  homme-machine  optimale.  L’Oeil  IR  a  subi  un  premier  test  en  vol  k  bord  de  V Avion  de  Recherche  pour  les  Systemes  £volu6s 
(heticoptere  Bell  412)  du  Laboratoire  de  Recherche  en  Vol  du  ConsefI  National  de  Recherche  du  Canada.  Ce  rapport  d6crit  le  prototype  et  I’approche  suivie 
pour  sa  conception,  presente  quelques  r6sultats  du  test  en  vol,  souligne  ses  forces  et  ses  d6ficiences,  et  suggdre  des  amelioration  pour  un  future  syst&me 
plus  avanc6. 
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Search  &  Rescue,  SAR,  Infrared  Camera,  Surveillance,  Multi  FOV 
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